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Achieving the Dream 

Community Colleges Count 

Achieving the Dream: Community Colleges Count is a national 
nonprofit that helps more community college students succeed, 
particularly students of color and low-income students. The 
organization works on multiple fronts-including efforts on 
campuses and in research, public engagement and public policy-and 
emphasizes the use of data to drive change. Launched as an initiative 
in 2004 with funding provided by Lumina Foundation for Education, 
Achieving the Dream is built on the belief that broad institutional 
change, informed by student achievement data, is critical to 
significantly improving student success rates. Today, Achieving 
the Dream's network includes 130 institutions in 24 states and the 
District of Columbia, reaching more than one million students. 
Achieving the Dream continues to work closely with founding 
partners: the American Association of Community Colleges (AACC); 
the Community College Leadership Program at the University of 
Texas-Austin (CCLP); the Community College Research Center, 
Teachers College, Columbia University (CCRC); Jobs for the Future 
(JFF); MDC; MDRC; and Public Agenda. 
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Jobs for the Future develops, implements, and promotes new 
education and workforce strategies that help communities, states, 
and the nation compete in a global economy. In 200 communities 
in 41 states, JFF improves the pathways leading from high school to 
college to family-sustaining careers. JFF coordinates the effort to 
improve policies in 16 Achieving the Dream states and co-leads the 
national policy effort. 
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TIME TO COMPLETION 



Many barriers that extend students' time to completion can be 
addressed through improvements in policy and practice. States, 
systems, and institutions have begun to experiment with new ideas 
that hold real promise for promoting timely completion. Building on 
those experiments, Jobs for the Future's Time to Completion project 
has two goals: expanding what we know about time to completion 
through research and analysis; and advocating for policies and 
practices leading to more timely completion for a greater number 
of students. Time to Completion is funded by Lumina Foundation 
for Education, which is committed to ensuring that 60 percent of 
Americans are college educated by 2025. 
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Education 
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The Institute for Higher Education Leadership & Policy at Sacramento 
State produces research and services for policymakers, practitioners, 
and educators to enhance leadership and policy for higher education 
in California and the nation. 
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OVERVIEW 



As educators, government officials, and foundation leaders have embraced the agenda of dramatically increasing college 
success and credential completion, they have come to understand the need for better data on student outcomes to guide 
the improvement efforts of institutions, systems, and states. The data on degree completion and other final outcomes are 
too little, too late if the ultimate goal is improving rather than just reporting outcomes. 

What is needed are good comparative data on intermediate steps along the way to completion that are associated with 
earning degrees. Such measures are likely to include basic skills acquisition and the completion of a specific number of 
credits or particular gateway courses. In Taking the Next Step, Jeremy Offenstein and Nancy Shulock assess the state of the 
emerging field of defining, measuring, reporting, and rewarding student progress in achieving "intermediate measures of 
success." 

Taking the Next Step distinguishes between milestones that must be attained in order to get to completion and success 
indicators that increase a student's chances of completion. The authors review 11 cases of the use of intermediate measures 
by multistate, single-state or single-system, and multi-institution initiatives, noting differences in approaches, definitions, 
and uses of "milestones" data. 

Taking the Next Step concludes with clear advice for the "next generation" of efforts to collect and report data on 
intermediate measures. The goal is to encourage common practices and definitions, as well as more thoughtful and 
effective uses of these data for institutional improvement, policy reform, performance funding, and accountability 
purposes. 
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THIS REPORT 
EXAMINES THE 
CONCEPT OF 
MILESTONES, 
EXPLORES THE 
WAYS THEY ARE 
BEING USED, AND 
CONSIDERS HOW 
THEY CAN BE USED 
MOST EFFECTIVELY. 



BETTER DATA FORA MORE 
COMPETITIVE FUTURE 



Increasing postsecondary attainment is an urgent national priority for ensuring long-term economic 
growth and prosperity. Most states are struggling to educate younger generations in order to fill 
the growing number of jobs that reguire educational credentialing beyond high school. More and 
more nations are surpassing the United States in the proportions of their populations with college 
degrees. Following several years of substantial investment by some of the nation's major philanthropic 
foundations, President Barack Obama has elevated the issue toward the top of the national agenda, 
calling for the United States to once again lead the world in postsecondary education levels. 

Educators, government officials, and foundations have begun to recognize that better data are 
essential to the strategy for meeting this challenge. The current means of measuring and accounting 
for outcomes in community and technical colleges and four-year institutions are deficient. Among 
the weaknesses is a lack of data to help postsecondary systems and institutions learn how to improve 
student outcomes by changing their interactions with students or redesigning program and service 
delivery. Knowing that far too few students complete their college education is not enough: we need to 
know where and why they get hung up in order to take appropriate steps. 

An emerging strategy for using data to improve postsecondary outcomes is to measure the patterns 
by which students reach and move through intermediate stages-often called ''milestones"-as they 
progress toward completing degrees or other postsecondary credentials. These approaches also track 
students' academic behaviors and link them to the achievement of milestones as a basis for examining 
which groups of students are (or are not) exhibiting behaviors that increase one's chance of succeeding. 
The promise of this approach is not only does it show where students stop progressing, but it also 
suggests why they have stopped and can help institutions identify how to improve student progress. 

Obtaining better data to support student success strategies is of particular value to community colleges 
because, compared to four-year institutions, they serve students with more risk factors for dropping 
out and with a more varied set of academic needs and goals. Traditional measures-retention and 
graduation-ignore the public value provided by community colleges when they help students earn GEDs, 
achieve basic literacy, or attain skills that are valued in the workplace (Morris et ai. 2005). Intermediate 
milestones, in addition to improving graduation rates, provide a means for two-year institutions to 
document and improve valuable student achievements short of completion. 

Although the milestone concept appeared only recently, a number of states and postsecondary 
institutions are applying it, some on their own and some in partnerships. Several of these initiatives 
are under development as the parties involved refine their metrics. New instances of milestone use are 
sure to follow soon as demand for higher graduation rates builds. As the milestone concept develops 
and expands, it is critical that these efforts be well-conceived from the outset. Once reporting systems 
are in place and institutions gear up to provide a set of metrics to system offices, governors, or state 
legislatures, it will be difficult to refine the approaches. In the interest of providing timely guidance, this 
report examines the concept of milestones, explores the ways they are being used, and considers how 
they can be used most effectively. 
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CONCEPTUAL FRAMEWORKS 



In 2006, Peter Ewell, as part of a partnership among states participating in the Ford Foundation Bridges to Opportunity and 
the Lumina Foundation for Education-funded Achieving the Dream initiatives, provided the conceptual basis for reporting 
intermediate measures with a continuum for tracking students from pre-college coursework to college completion (Ewell 
2006). Along the continuum are several "milestone events" through which students progress en route to completing 
a Bachelor's degree. Superimposed on the milestone events are several key measures of intermediate progress, such 
as the developmental completion rate (the share of students who start developmental work and complete it) and the 
"college path" completion rate (the share of students who complete a term of college-level work and end up completing an 
Associate's degree). 

FIGURE 1. "MILESTONE EVENTS” IN A STUDENT ENROLLMENT PATHWAY 
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Building on Ewell's framework, the Community College Research Center published a research tool for applying a milestone 
framework to data from the Washington State Board for Community and Technical Colleges (Leinbach & Jenkins 2008). 
This research groups intermediate measures into two types: milestones and momentum points. 

Milestones are measurable educational achievements that vary by students' levels of preparation and goals. In this 
framework, the milestones identified are: 

> Completion of pre-collegiate coursework; 

> Transition into college-level coursework; 

> Transfer to a four-year university; and 

> Completion of an educational program or apprenticeship. 
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Momentum points are measurable educational attainments that predict completion of a milestone. In other words, 
students who attain momentum points are more likely to achieve the milestones than students who do not. The momentum 
points identified are: 

> Completion of one pre-collegiate course; 

> Completion of a career exploration or introduction course; 

> Completion of one college-level gatekeeper math course; 

> Completion of a college-level gatekeeper English course; 

> Completion of 15 college-level credits; 

> Completion of 30 college-level credits; 

> Completion of 30 college-level credits in one year; 

> Completion of 15 vocational credits; 

> Completion of 30 vocational credits; and 

> Completion of 30 vocational credits in one year. 

Using this framework, the researchers tracked the number of students who completed each of the milestones for different 
enrollment groups, such as students starting at pre-collegiate levels, college-level students, and workforce training 
students. The analysis shows that students who attain the momentum points are more likely to complete a milestone. 

Steps to Success, a research report by the Institute for Higher Education Leadership & Policy (IHELP), applies a similar 
framework to data from the California Community College system (Moore, Shulock, & Offenstein 2009). Like the Community 
College Research Center study, this report classifies measures into two types: intermediate achievements, also referred 
to as "milestones," and measurable academic behaviors that predict completion, referred to as "success indicators." The 
success indicators include a variety of measures drawn from the student success literature, such as: 

> Completing a specific number of credits in a specified period of time; 

> Completing gateway courses in a specified period of time; 

> Completing a student success course; 

> Earning summer credits; 

> Enrolling continuously; and 

> Avoiding excessive course withdrawal and late registration for classes. 

The IHELP report shows that students who engaged in these behaviors were more likely to complete a degree, certificate, 
or transfer. It also illustrates how additional analysis of the success indicators could point to possible changes in policy and 
practice that would improve student outcomes. A subseguent joint publication by IHELP and the Education Trust, Advancing 
by Degrees, applied the same framework to the State University System of Florida and the California Community Colleges to 
demonstrate the utility of the framework for four-year as well as two-year institutions (Offenstein, Moore, & Shulock 2010). 
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An issue brief by the National Governors Association argues that state accountability systems should measure milestone 
achievement in addition to graduation rates; a sole focus on graduation rates creates an incentive for institutions to enroll 
students most likely to succeed and to underserve students from disadvantaged backgrounds (Bearer-Friend 2009). The 
NGA brief also notes that by providing more information about where students tend to get stalled, tracking milestones 
helps policymakers learn how they might improve student outcomes. It focuses on three types of milestones: remediation 
milestones, retention milestones, and attainment milestones. The report suggests that states track remediation milestones 
that predict eventual degree completion. It also identifies pass rates in remedial and core courses and advancement from 
remedial to college-level courses as key milestones. The key retention milestone identified is transfer from a two-year to 
a four-year institution, but the report recommends tracking semester-to-semester retention, full-time enrollment, and 
continuous enrollment. For attainment milestones, it recommends tracking attainment of certificates and Associate's 
and Bachelor's degrees-and disaggregating performance on these measures by institution and by several student 
subpopulations: 

> Part-time and full-time students; 

> Transfer students; 

> Students beginning in remedial courses; 

> Students with limited English proficiency; 

> Pell-eligible students; 

> Underrepresented ethnic minorities; 

> Students in science, technology, engineering, and mathematics (STEM) fields; 

> Students above age 21 when first enrolled; and 

> Students with disabilities. 

As these frameworks have influenced the implementation of the milestone concept across the country, it is instructive to 
see how they compare conceptually, before taking a look at the actual applications of milestones by institutions and states. 
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TWO TYPES OF MEASURES 



The frameworks profiled above suggest a wide variety of measures, reflecting different priorities and interests. Moreover, 
they describe two gualitatively different types of measures, although the frameworks themselves do not always make the 
distinction. One type is the set of intermediate points that students must reach if they are going to complete a program or 
degree. These include achievements like completing a college-level math course and 15 college-level units. These necessary 
achievements differ depending on a student's goal and starting point. For example, transfer is a necessary achievement for 
a baccalaureate-seeking student who begins at a community college, but not for a student who begins at a university or for 
a community college student who seeks a workplace certificate. However, for all students with a similar program goal, the 
necessary achievements are the same and must be attained for successful completion of the program. 

The second type of intermediate measure tracks academic behaviors that increase a student's chances of completion but 
that are not strictly required in order to finish an academic program. Academic behaviors that have been shown to correlate 
with success and that fall into this category of measure include completing college math within the first two years, enrolling 
in a summer session, and minimizing late registration and course withdrawals. Again, these can vary for different types 
of students. For example, completing a college-level math course within a set number of terms may be more important in 
increasing chances of success for younger students than for older students (Calcagno et al. 2007). 



INCONSISTENT USE OF MILESTONE TERMINOLOGY 



While the frameworks use the term "milestones," there are inconsistencies among them in how the term is used, and in how 
they treat the second type of measure: those that correlate with or predict success but are not strictly required for success. 

The Community College Research Center separates student outcomes that are recognizable academic achievements 
from those that predict those achievements, calling the former "milestones" and the latter "momentum points." However, 
CCRC includes as milestones some measures that fall into the "correlates of success" category because they have a time 
dimension. As an example, completing a college-level math course within one year is classified as a milestone by CCRC, but a 
student could delay college-level math to the second year or beyond and still complete a degree. Conversely, CCRC classifies 
completion of 15 or 30 credits as a momentum point because it predicts the completion of milestones-even though this is a 
necessary achievement on the road to degree completion. 
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MILESTONES: SPECIFIC 
EDUCATIONAL ACHIEVEMENTS 
THAT STUDENTS MUST 
ACCOMPLISH TO GET FROM 
THEIR BEGINNING POINT 
TO COMPLETION OF AN 
EDUCATIONAL PROGRAM, AS 
WELL AS THOSE ACHIEVEMENTS 
THAT MARK THE END OF THE 
EDUCATIONAL PROGRAM. 



IHELP's Steps to Success framework uses ‘‘milestones" narrowly to refer to 
necessary intermediate educational achievements. It uses the term "success 
indicators" for academic behaviors that predict success but are not strictly 
required in order to succeed. At the other extreme, the NGA report uses the term 
"milestone" in reference to both kinds of measures without attempting to make a 
distinction. Included among their milestones are measures of academic behaviors 
that correlate with success but are not strictly required in order to succeed (e.g., 
continuous enrollment, advancement from part-time to full-time status). 

This paper uses the terms "milestone" and "success indicator" to differentiate 
between the two types of measures. 



SUCCESS INDICATORS: 

ACADEMIC BEHAVIORS THAT 
INCREASE A STUDENT’S 
CHANCES OF COMPLETION BUT 
ARE NOT STRICTLY REQUIRED IN 
ORDER TO FINISH AN ACADEMIC 
PROGRAM. 



THE IMPORTANCE OF DISAGGREGATING DATA 



Despite these differences, all of the frameworks highlighted above emphasize 
the importance of disaggregating data by sub-populations of students. The 
CCRC report emphasizes disaggregating the data and tailoring the intermediate 
measures for different groups based on whether their goal is obtaining a 
vocational credential or transferring. The National Governors Association 
recommends reporting milestone achievements for students in STEM fields. 
Multiple reports recommend disaggregating by student demographic groups, 
level of preparation, and whether students attend part time or full time. 
Disaggregating by level of preparation is particularly important given the 
difficulty that poorly prepared students have completing developmental 
education and entering and succeeding in college-level coursework. 
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INTERMEDIATE MEASURES INACTION 



The use of data to examine student and institutional outcomes is certainly not new. Institutional researchers have been 
engaged in such analyses for decades. What is new is the conscientious application of milestone analysis to improve 
ultimate outcomes-and the scale of several initiatives that engage numerous states and/or postsecondary systems in data- 
rich efforts to change institutions and public policies. 

Here we review 11 cases of milestone application that are of significant scale, as judged both by the number of participants 
and the potential to influence public policy. Some of these are major, multistate initiatives that arguably have the greatest 
opportunity to influence the practice of using intermediate measures. Some are limited to one state or postsecondary 
system; others involve groups of institutions, either within a state or across states. All are furthering the national dialogue 
about the use of intermediate measures to improve postsecondary outcomes across institutions, systems, and states in the 
interest of enhancing students' social and economic prospects. 

In every case, intermediate measures-whether proposed or in place-go well beyond the traditional measures of retention, 
transfer, and graduation. We describe each case and summarize its key features with respect to: the types of measures 
used; the unit of analysis; and the principal uses for the data. 



MULTISTATE INITIATIVES 

ACHIEVING THE DREAM CROSS-STATE DATA WORK GROUP 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts (disaggregated by student characteristics) 


PRINCIPAL USES FOR DATA 


Institutional improvement; state and system policy reform 



Achieving the Dream (ATD) is the first initiative to apply the milestone concept to improving policy, practice, and outcomes 
in community colleges across the country. ATD's Cross-State Data Work Group began with participation by postsecondary 
systems from Connecticut, Florida, North Carolina, Ohio, Texas, and Virginia, and is coordinated by Jobs for the Future 
(Brown 2009). 

The effort focused initially on developing a more accurate set of completion measures (e.g., part-time students) that 
would be an alternative to what was available in the Integrated Postsecondary Education Data System (IPEDS) for use 
in community college accountability reporting. Subsequently, the work group moved on to developing intermediate 
benchmarks that provide additional information for improving student outcomes. Participants have benefitted 
tremendously from the collaborative process of developing and applying the measures of student and system performance- 
particularly in view of the work group's interest in the effects of state policy on student outcomes. For example, degree 
completion rates are considerably higher in Florida, where students who complete an Associate's degree are granted 
admission to a university, than they are in Texas, where there is less incentive for students to complete an Associate's 
degree before transferring (Jobs for the Future 2008). 

The measures developed by the Cross-State Work Group are also being used in the Developmental Education Initiative 
funded by the Bill & Melinda Gates Foundation. This three-year effort, begun in 2009, is identifying and creating programs 
that increase the number of community college students who complete preparatory classes and advance to college-level 
studies. With a focus on policy and practice spanning institutional, system, and state levels, Achieving the Dream and the 
Developmental Education Initiative are the nation's most comprehensive data-driven community college reform efforts. 
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ACCESS TO SUCCESS INITIATIVE 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts (disaggregated by student characteristics) 


PRINCIPAL USES FOR DATA 


Institutional improvement; system policy reform 



Another effort to use intermediate measures of student outcomes and academic behaviors involves a group of 
postsecondary systems participating in the Access to Success Initiative spearheaded by the National Association of System 
Heads (NASH) and The Education Trust. Access to Success brings together 24 public higher education systems that have 
pledged to cut the college-going and graduation gaps for low-income and minority students in half by 2015. The initiative 
aims to build capacity at the system level to lead change efforts and stimulate institutional improvement through data- 
driven changes to system policy and practice. 

The initiative has developed a baseline report of access and success metrics for two-year and four-year institutions. 

The metrics focus on "leading indicators" that correlate with student success (what this report calls success indicators). 
Although the effort is in the early stages, the sharing of data and discussions that this initiative is advancing are 
of potentially great value because NASH member institutions collectively serve 40 percent of the nation's public 
postsecondary students. Its impact on institutions that are not organized into systems (e.g., Arizona's community colleges) 
would be indirect. 



COMPLETE COLLEGE AMERICA 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts and annual counts (disaggregated by student characteristics) 


PRINCIPAL USES FOR DATA 


State accountability; state policy reform 



Established in 2009 with the support of five national foundations, Complete College America is a national nonprofit working 
to significantly increase the number of Americans with a college degree or credential of value and to close attainment 
gaps for traditionally underrepresented populations. Starting with the premise that states have the authority to shape 
postsecondary institutions and the capacity to lead systematic change, CCA's principal tactic for achieving the intended 
results is state-level policy change. Governors and other state leaders must commit to being accountable for selected 
outcomes and to report a common set of agreed upon measures that include intermediate benchmarks (National Center 
for Higher Education Management Systems n.d.; National Governors Association 2010). Twenty-two states have made this 
commitment. 

The specific measures and reporting reguirements are under development and reporting has not yet begun. The intent 
is that completion data will be reported at the state and institutional levels, while the intermediate measures will be 
reported at the institutional level only. The National Governors Association has adopted the same set of metrics and lent 
its name to the effort-which is now called "NGA Common Metrics." The CCA and NGA metrics are grouped into progress 
metrics and outcome metrics. This grouping is based on when in the course of an educational program students reach the 
metrics, not whether or not the metrics are necessary for them to complete a program. This contrasts with our approach, 
which distinguishes between intermediate points that students must pass through and academic behaviors that predict 
completion but are not necessary for it. 
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SINGLE-STATE OR SINGLE-SYSTEM INITIATIVES 

THE WASHINGTON STATE STUDENT ACHIEVEMENT INITIATIVE 



TYPES OF MEASURES 


Milestones 


UNIT OF ANALYSIS 


Annual counts 


PRINCIPAL USES FOR DATA 


State accountability; performance funding; institutional improvement 



Arguably the most well known state effort to measure intermediate student outcomes is Washington's Student Achievement 
Initiative.' Led by the Washington State Board for Community and Technical Colleges, the SAI is based on research by the 
Community College Research Center and tied to the policy goal of increasing educational attainment for all Washingtonians. 
The initiative measures six sets of "achievement points,” ranging from increasing basic skills to completing degrees and 
certificates. Colleges earn points based on the number of students who reach the specified achievements. Colleges receive 
funding in addition to their regular state appropriations for increases in the number of achievement points they earn in 
a year. 

The SAI is the first instance of a system-wide performance funding policy designed around a set of intermediate measures. 
Although the funding allotted for the initiative is less than 1 percent of the system's budget, there is interest among the 
state's policy leaders in increasing the share of funding awarded based on achievement point performance. 

OHIO PERFORMANCE FUNDING 



TYPES OF MEASURES 


Milestones 


UNIT OF ANALYSIS 


Annual counts 


PRINCIPAL USES FOR DATA 


State accountability; performance funding; institutional improvement 



The Ohio Board of Regents recently initiated a performance funding effort that applies across all sectors of public higher 
education but uses a different approach for each sector. The approach for four-year institutions does not involve the use of 
intermediate measures. For community colleges, Ohio uses a set of intermediate measures similar to those in Washington, 
with additions that the board feels are important to track. 2 This framework grew out of the state's participation in the 
Achieving the Dream Cross-State Data Work Group and from the work done in Washington. 

As in Washington, Ohio colleges will accumulate "success points" based on the number of students who attain the 
measured achievements. A key difference from the Washington Student Achievement initiative is that Ohio colleges will 
receive a portion of their basic allocation based on their shares of the total momentum points earned. The portion of 
funding generated by points will begin at 5 percent in the 2011 fiscal year, with plans to increase that share to 30 percent. In 
Washington, the base-funding model is unchanged and colleges earn performance funds on top of base funding. As noted, 
the portion of performance funding is much less than what is planned in Ohio. 

CITY UNIVERSITY OF NEW YORK 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts and annual rate counts 


PRINCIPAL USES FOR DATA 


Institutional improvement 



The City University of New York also uses intermediate measures of student outcomes and academic behaviors as part of 
a broad array of indicators in its Performance Management Process (City University of New York n.d.). The Performance 
Management Process, an important component of the university system's planning process, is intended to provide clarity 
about the institutions' priorities and to recognize performance in meeting those priorities. Most of the measures are applied 
to both the two- and four-year institutions in the CUNY system, although some are applicable only to one type of institution. 
Financial incentives are tied to performance on all of the indicators when funds are available. 



JOBS FOR THE FUTURE 



9 



TENNESSEE PERFORMANCE FUNDING 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts and annual rates 


PRINCIPAL USES FOR DATA 


State accountability; performance funding; institutional improvement 



The Tennessee Higher Education Commission's performance funding measures for the 2005-10 cycle and, more specifically, 
the student persistence measures for community colleges, include some intermediate measures of student progress. 3 In 
this accountability system, all community colleges are assessed on fall-to-fall persistence and degree completion rates. 
Additionally, colleges choose four other measures from a set of eight, some of which are intermediate outcome measures 
(e.g., proportion of students who successfully complete their developmental course with a grade of A, B, C or P) and enroll 
in a related college-level course; the proportion of students who successfully complete college algebra). Institutions' 
performance on these measures is judged through comparisons to data from peer institutions that report data to the 
National Community College Benchmarking Project. 



MULTI-INSTITUTIONAL INITIATIVES 

NATIONAL COMMUNITY COLLEGE BENCHMARKING PROJECT 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts and annual rates 


PRINCIPAL USES FOR DATA 


Institutional improvement 



Two hundred and ten participating colleges report performance data on some of the 130 benchmarks used in this project, 
although no college reports on all of the benchmarks. 4 Participating colleges receive data in exchange for their own 
information and a small fee. The data from the project are not publicly available. 

The importance of this project is its potential influence on state accountability reporting and the capacity of institutions to 
track performance on intermediate measures. Tennessee's adoption of its performance measures is a good example of how 
the availability of data can lead states to adopt certain measures in their own accountability systems. Tennessee chose its 
performance metrics in part because of the availability of comparative data from the benchmarking project. Thus, although 
the benchmarking project focuses primarily on individual institutional improvement, it can affect state policy and certainly 
can promote the expanded use of data and of intermediate measures across the country. 



CALIFORNIA LEADERSHIP ALLIANCE FOR STUDENT SUCCESS 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts and annual rates 


PRINCIPAL USES FOR DATA 


Institutional improvement 



The California Leadership Alliance for Student Success is an example of a project focusing on a subset of institutions 
within a single postsecondary system with the potential to affect state policy (Center for Community College Student 
Engagement 2009). CLASS, a collective effort of 12 community colleges focusing on leadership strategies for improving 
student success. Project staff, funded by foundations, bring teams of college leaders together to discuss student success 
issues. A key component of the work of these college teams is the provision and analysis of data on intermediate measures 
(e.g., completion of the first collegiate-level courses in English composition, U.S. history, and college algebra) and academic 
behaviors (e.g., share of courses completed successfully). Institutions use a tool to run a set of analyses on data that relate 
to the topic of discussion. Although it only involves 12 colleges from a system of 112, the examination of the data in relation 
to student success could have important system-level and state policy implications that would ultimately affect student 
success at all California community colleges. 
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AMERICAN ASSOCIATION OF COMMUNITY COLLEGES’ VOLUNTARY FRAMEWORK OF 
ACCOUNTABILITY 



TYPES OF MEASURES 


Milestones and success indicators 


UNIT OF ANALYSIS 


Cohorts 


PRINCIPAL USES FOR DATA 


Knowledge building in support of institutional improvement; state accountability; 
state policy 



This project follows closely upon the Voluntary System of Accountability begun in 2007 by a group of four-year universities 
under the sponsorship of two of their membership associations. As with VSA, this initiative appears to be aimed at 
demonstrating accountability to state policymakers through means that the colleges themselves determine are appropriate. 
The definitions and instructions for calculating the measures will be reported in fall 2011. In view of the important function 
that intermediate measures are serving for community colleges, it is likely that intermediate measures will be prominent in 
this project as well. 



BILL & MELINDA GATES FOUNDATION STUDENT PROGRESS AND COMPLETION 
MEASURES FOR GRANTEES 



TYPES OF MEASURES 


TBA 


UNIT OF ANALYSIS 


TBA 


PRINCIPAL USES FOR DATA 


State accountability 



The Gates Foundation is drafting a set of measures that it plans ultimately to ask all grantees in its community college 
portfolio to report. The intent is to foster consistency among grantees about how to measure results, accelerate knowledge 
building in the field by facilitating comparisons across projects, and contribute to the growing momentum toward a common 
set of metrics to assess student progress and completion. Drawing from some of the above cases, the current metrics 
include a wide variety of success indicators and milestones. With so many grantees involved in other initiatives that require 
reporting, the foundation hopes to design a template that will meet its somewhat different needs (which include internal 
evaluation) while minimizing reporting burdens on its grantees. 
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CONSIDERATIONS FOR USING 
INTERMEDIATE MEASURES TO INCREASE 
STUDENT SUCCESS 



The range of efforts reviewed above demonstrates that the first generation of milestone application promises to enrich our 
understanding of student transitions, progress, and completion. The following discussion is intended to influence the next 
stage of development by suggesting how these efforts might coalesce around some common practices in order to be most 
effective and increase alignment across these efforts. 



1. DISTINGUISHING MILESTONES FROM SUCCESS INDICATORS CAN HELP 
MATCH DATA WITH USERS AND PURPOSES 



In most cases where intermediate measures are being reported and used, no explicit distinction is made between 
milestones and success indicators. Making a distinction could be helpful because the two types of measures are best suited 
for use by different stakeholders for somewhat different purposes. 

While the ultimate purpose of tracking and acting upon better data is to improve student outcomes; institutional leaders, 
postsecondary system leaders, and state-level policy actors play different roles in accomplishing this common purpose. 
Institutional leaders need to examine student-level data in detail to determine where student progress is stalling and why- 
the latter so that they can adjust institutional practices (e.g., the operation of academic advising; the class schedule; the 
developmental math curriculum). State policymakers need to know a bigger picture: are they getting the outcomes they 
need from public investments in postsecondary education and policies? They typically do not need to dig as deeply into 
data as institutional leaders do. 

Postsecondary system leaders occupy a key place that spans institutional practice and state policy. Particularly in states 
where postsecondary systems have authority over institutions and chief responsibility for state governmental relations, 
system leaders are accountable to state policymakers for shaping system policies and influencing campus practices to 
deliver the desired outcomes. They are accountable to their own institutions to represent the needs of the system at the 
state level in order to seek policy reforms and appropriate resource investments. To fulfill these responsibilities, they need 
to know where and why students are stalling, what might be done about it, and what policy changes at the system and state 
level are needed. 
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Table 1 shows that the different primary roles in the effort to increase student success have implications for the types of 
measures that are most appropriate at each level. 



TABLE 1. PRIMARY USE OF INTERMEDIATE MEASURES, BY LEVEL OF ACTOR 



ACTOR 


PRINCIPAL INTEREST 


TYPE OF MEASURE 


PRIMARY USE 


Institutional leaders 


Where is student progress 
stalling and how can campus 
practices be changed to 
improve success? 


Milestones and Success 
Indicators 


Institutional Improvement 


Postsecondary system 
leaders 


Flow can the system's 
institutions collectively 
deliver the results that the 
state wants and needs? 


Milestones and Success 
Indicators 


Institutional Improvement; 
State Accountability; 

State and System Policy 


State-level policymakers 


Flow well are the state's 
investments and policy 
design delivering the results 
that the state needs? 


Milestones 


State Policy 



Milestones are the intermediate and final outcome measures that are most useful for state-level actors: they help 
lawmakers understand whether state goals are being met. Lawmakers need outcomes information to help them decide how 
best to invest state resources across the scope of state programs. They have less need for information on how institutions 
have produced the outcomes (i.e., the kinds of success indicators included in many of the cases reviewed here). For example, 
state lawmakers probably do not need to know whether community colleges have introduced mandatory student success 
courses or if universities reguire students to declare majors by the time they have earned 60 credits. Unless information 
is directly relevant to a state-level policy decision, it seems unnecessary to include it in a state-level reporting system, 
and doing so could detract from the simplicity of such a system. In fact, reporting on success indicators could encourage 
lawmakers to micromanage institutions, rather than to hold them accountable for outcomes and give them the flexibility to 
produce those outcomes as they best know how. 

To be most useful to state lawmakers, a reporting system should include milestones that reflect the state's specific goals. 
States would likely have some milestones reflecting their particular goals. For example, states that want to increase 
the use of the two-year sector as a gateway to Bachelor's degrees could emphasize such milestones as completion of a 
lower-division general education pattern or transfer from a community college to a four-year institution. States that are 
particularly focused on increasing the employability of undereducated adults could track skill gains in adult basic education 
and completion of a year's worth of credits (the level of credits associated with an increase in earnings) (Grubb 2002). If a 
state is concerned with developing a knowledge-based, high-technology economy, it could track the completion of gateway 
and advanced mathematics courses, community college student transfers to universities in selected fields, and the award 
of Bachelor's and graduate degrees in STEM fields. As a final example, states with growing performance gaps among 
minority populations could track transitions from developmental to college-level coursework for underrepresented minority 
populations and the subseguent earning of college degrees. 

At the institution and system levels, both milestones and success indicators are essential types of intermediate measures 
to collect and analyze. It is the promise of the pairing of these two types of measures that is driving the interest in 
intermediate measures across the country. Milestones can be tracked to help institutions see where student progress gets 
stalled and to focus institutional attention at the troublesome transition points. Success indicators can then be examined to 
explore why students are not making better progress. 
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Examining milestones alone can certainly be instructive to institutions and systems. For example, milestone data showing 
that few students make the transition from adult basic education to college-level instruction could lead institutions 
to better align the curriculum between sectors, develop new tools for assessing student skills, or change instructional 
practices. As another example, data showing high levels of attrition after students have completed a year's worth of college 
credit might lead to the expansion of "first-year experience" interventions beyond the first year. And data showing that 
most students never complete college-level math can galvanize efforts to improve math instruction. 

Examining success indicators alongside milestones can deepen understanding of the problem and help suggest what 
specific interventions may be warranted. Success indicators measure things that students do while pursuing the completion 
of a program. For example, students attend full time (or part time), they enroll in remedial work in their first term (or delay 
remediation), they register for their courses on time (or enroll late for many courses), and they attempt college-level math 
by their second year (or put off taking math). Because these academic behaviors have been documented in research to 
correlate with completion, they are of diagnostic value and can help guide system and institutional policy and practice to 
improve outcomes for students. 

As shown in Table 2 on page 15 using California data, a larger share of students who complete college-level math within two 
years and students who earn at least 20 credits in their first year achieve each of the milestones assessed. For example, 

61.1 percent of students who completed college-level math within two years completed a certificate or Associate's degree or 
transferred within seven years compared to 22.0 percent who did not complete math within two years. These data point to 
strategies (e.g., better advisement; accelerated developmental education) that help students complete college-level math 
early as a way to increase completion rates. Success indicators can be identified through a review of existing research 
on predictors of student success or through finding correlations of milestone attainment in the analysis of system or 
institutional data. 
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TABLE 2. MILESTONE ACHIEVEMENT BY SUCCESS INDICATOR ATTAINMENT IN A 
COHORT OF CALIFORNIA COMMUNITY COLLEGE STUDENTS 





RETENTION TO 2ND TERM 


RETENTION TO 2ND YEAR 


EARNED 12+ COLLEGE- 
LEVEL CREDITS 


EARNED 30+ COLLEGE- 
LEVEL CREDITS 


COMPLETED TRANSFER 
CURRICULUM 


EARNED CERTIFICATE 


EARNED ASSOCIATE’S 
DEGREE 


TRANSFERRED 


ANY COMPLETION 


COLLEGE-LEVEL MATH 


Completed 
within 2 years 
(21%) 


92.8% 


86.7% 


95.9% 


83.4% 


50.4% 


3.2% 


20.6% 


53.1% 


61.1% 


Did not 
complete 
within 2 years 
(79%) 


70.1% 


52.2% 


55.4% 


33.3% 


9.9% 


3.4% 


5.3% 


16.9% 


22.0% 


FIRST-YEAR CREDITS 


Earned 20+ 
credits (24%) 


99.3% 


89.0% 


97.7% 


86.5% 


44.2% 


6.2% 


20.6% 


46.4% 


57.9% 


Did not earn 
20 credits 
(76%) 


66.0% 


48.3% 


51.1% 


27.7% 


8.1% 


2.4% 


3.9% 


15.7% 


19.4% 


Source: Moore, Shulock, & Offenstein (2009) 



Institutions and systems can dig deeper into the data to understand more fully what changes to policies and practice may 
affect student success. Figure 2 on page 16 shows how further analysis can begin to break down the reasons why students 
fail to accumulate 20 or more credits in their first year in a cohort of students in the California Community College system. 
Most students in this cohort did not complete 20 credits because they did not attempt that many credits. This finding points 
to the potential of financial aid policy and practices for increasing the number of students who reach the 20-credits-in- 
the-first-year success indicator. Perhaps more surprising is that nearly one-quarter of the students who did not reach the 
20+ credit level actually attempted well over 20 credits, on average, but dropped or failed a large portion of those classes. 
This finding suggests a different set of potential interventions related to policies on course drops and repeats as well as 
practices related to academic assistance services like tutoring and early alert programs. 
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FIGURE 2. PATTERNS RELATED TO EARLY CREDIT ACCUMULATION 




> Policies related to course 

dropping, course repeats, and use 
of summer terms 



> Practices related to tutoring 
and other academic assistance 
services 

There is a wide variety of success indicators and systems; institutions need not track all of them in their routine monitoring 
processes. Just as state-level reporting systems should be simply structured around key policy goals, institutional reporting 
should coalesce around those indicators that best illuminate patterns of student progress and success. Some of the 
measures identified in the research literature as predicting student success are more significant than others, and some 
duplicate the information provided by others (Moore & Shulock 2009). For example, whether a system tracks completion of 
20 credits in a year or the first-year credit completion ratio is probably not going to make a large difference in the ability 
to diagnose completion problems. 5 These two measures are so highly correlated that tracking one is probably sufficient. 

As another example, an institution may find that tracking first-year credit completion ratios and first-year GPA provides 
no additional information than tracking one or the other. To make the best use of their analytical resources, colleges and 
systems should track a few measures that capture a variety of academic behaviors rather than track multiple variations of 
essentially the same measures of academic behavior. 

Based on a review of the research literature, key success indicators to track are measures of progress through remediation, 
gateway course completion, credit accumulation, and related academic behaviors (Moore & Shulock 2009). Colleges and 
systems can use these measures to change policy and practice in order to improve outcomes for students because they 



16 



TAKING THE NEXT STEP 



COLLEGES AND SYSTEMS CAN 
USE THESE MEASURES TO 
CHANGE POLICY AND PRACTICE 
TO IMPROVE OUTCOMES FOR 
STUDENTS BECAUSE THEY 
PREDICT COMPLETION OF 
MILESTONES. 



predict completion of milestones. For example, institutions and systems may 
want to track the share of students enrolling right away in needed remediation. 

If a sizable number of underprepared students are delaying remediation, then 
the institution or system may want to create policies and engage in practices 
to ensure that all new students are assessed and directed to the appropriate 
coursework, and that enough sections of remedial courses are available for 
students who need them. Similarly, if colleges and systems tracking completion 
of college success courses find that a small share of students complete these 
courses, they can take steps to ensure that there are adeguate course offerings 
at convenient times and reguire or encourage students to take them. As a final 
example, when faced with many students who are dropping or failing a large 
share of their courses, colleges can develop early alert systems, improve tutoring 
services to provide more academic assistance, and enact policies to limit course 
dropping and repeats. 



2. MILESTONES ARE PROMISING COMPONENTS 
OF PERFORMANCE FUNDING APPROACHES 



The three state accountability efforts reviewed (Washington, Ohio, and 
Tennessee) tie funding to colleges' outcomes on intermediate measures. 
Washington and Ohio use similar benchmarks but have different methods of 
allocating dollars based on performance. In Tennessee, performance on the 
intermediate outcomes is a component of the formula for allocating performance 
funds. In addition to these state efforts, the CUNY system's performance 
management process includes financial incentives for performance on 
intermediate measures (specifically, presidential and executive raises and funds 
for professional activities such as travel, software purchases, and professional 
development). 

As states explore performance funding models to incorporate fiscal incentives 
for desirable outcomes, milestones are appropriately getting more attention. 
Earlier attempts at designing performance funding systems were flawed in part 
because they put too much emphasis on degree completion and overlooked the 
value of rewarding institutions for intermediate student achievements and for 
meeting other state priorities. Although performance funding is very much an 
evolving practice, the concept of milestones appears to have much to offer as 
states attempt to design effective and fair funding models. 
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3. COHORT TRACKING, DISAGGREGATED BY SUB-POPULATION, CAN BEST 
SUPPORT INTERVENTION STRATEGIES 



Some efforts to track milestones and success indicators follow cohorts of students, and some report annual activity in 
the form of numbers or rates of students who attain the outcomes or exhibit the academic behaviors of interest. Tracking 
cohorts of new students is generally preferable to tracking annual numbers or rates for purposes of accountability and 
institutional improvement. Cohort rates of students' milestone attainment are easily understood and well suited for 
accountability reporting because they assess productivity and institutional effectiveness. They are also easy to understand 
for the purposes of making changes to improve student outcomes. When examining the rate at which members of a cohort 
complete 30 college units, for example, it is easier for colleges and systems to think about the contextual factors that could 
explain the rate because they know when those students entered and that all students entered under the same institutional 
policies and practices. 

However, tracking cohorts is a more difficult task for institutional researchers. It reguires choices about students to 
include in the cohort (e.g., all students; degree-seeking students; students in need of remedial coursework) and the data 
necessary to construct the cohorts (e.g., course-taking behavior to determine degree-seekers). Another disadvantage of 
cohort tracking is the need to allow for a reasonable period of time before progress through the milestones and the cohort 
completion rate can be assessed. 

Annual activity, such as the number of students earning degrees in a year, is relatively easy to calculate. Institutions and 
systems using annual activity measures either report on the number of students who attain an outcome or engage in 
an academic behavior in a year, or calculate a rate by dividing this number by a population of students (e.g., all students 
enrolled). This approach is more suitable for accountability purposes than it is for institutional improvement, although it has 
limitations even for accountability reporting. First, counts of students who achieve milestones are only valuable if they are 
tied to specific state goals. For example, if a state determines that it needs 30,000 Bachelor's-degree earners a year, then 
the number of degree earners would indicate whether or not that goal was met. Absent an identified need, the number of 
students attaining outcomes in a given year is of little value because it is heavily driven by enrollment and does not gauge 
productivity or institutional effectiveness. Calculating annual rates is a better measure of productivity than just annual 
volume, but it too can be affected by year-to-year changes in enrollment. For example, the share of students who complete 
gatekeeper math out of all students enrolled can drop if the number of new underprepared students increases. 

Both cohort and annual counts can be most powerful if they are conducted separately for key sub-populations (e.g., 
underrepresented minorities; older students; underprepared students). Certainly, one of the nation's foremost challenges is 
closing the success gap across racial and ethnic groups. The in-depth analysis supported by milestone and success indicator 
data could be a significant boon in the concerted-but largely unsuccessful-efforts to date to close these gaps. Research 
has found that student attainment of milestones and the correlation to certain success indicators varies by student group 
(Moore et al. 2009). Identifying these differences can give systems and institutions insight into the completion issues faced 
by different groups and potential strategies to address them. 
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4. IMPROVED DATA SYSTEMS SHOULD BE A 
PRIORITY 



A COMMON PROBLEM IS THAT 
WHEN DATA ARE NOT USED FOR 
ANY MEANINGFUL PURPOSE, 
THERE IS NO INCENTIVE TO 
ENSURE THAT THOSE DATA ARE 
ACCURATE. 



Measuring and tracking milestones and success indicators requires a robust 
data system that includes student-level transcript data on course-level credits 
attempted and completed, with information on course titles and grades earned. 
Data on student demographic characteristics and academic backgrounds are 
also necessary in order to disaggregate reporting by student race or ethnicity, 
gender, age, income, and level of academic preparation. Even states and 
systems with well-developed data systems encounter difficulties in monitoring 
all of the intermediate outcomes and student characteristics that they would 
like. A common problem is that when data are not used for any meaningful 
purpose, there is no incentive to ensure that those data are accurate. Thus, 
when a system that routinely collects a wide variety of data tries to use them to 
analyze milestones and success indicators, it may find that much of the data are 
inaccurate or missing. 

Improving data systems can be costly. As a cost-savings action, measures are 
sometimes chosen because the data are available rather than because they 
are important. While this is a pragmatic way to begin to monitor student data, 
it is not advisable to build data and reporting systems around preexisting data 
elements. Institutions, systems, and states should strive ultimately to get data 
that will help them answer the questions they need answered in order to improve 
student success. 



Three common data challenges that systems and institutions face are: 

> Identifying which students need remediation; 

> Collecting data on family income levels; and 

> Tracking transfer students once they leave community college. 

Educating underprepared students is an ongoing challenge for higher education 
institutions and systems; community colleges are particularly challenged in 
this regard. To track progress in getting underprepared students ready for 
college-level work and completing college degrees or certificates, institutions 
and systems need good data about the preparation of entering students. Yet 
placement test scores and data on who is referred to remediation are often 
missing from data systems. Colleges and institutions frequently rely on whether 
or not students enroll in developmental coursework as a proxy for remedial 
need. However, this practice can lead to erroneous conclusions if many students 
in need of remedial coursework are not taking the classes. An analysis of 
California Community College data found that students who took remedial 
courses were about as likely to complete degrees as those who did not, despite 
the well-established relationship between academic preparation and success . 6 
That counterintuitive finding is likely the result of counting both well-prepared 
students and students needing but not taking remediation in the same group for 
lack of data on academic preparation. 
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Data on students' level of preparation and whether or not they were referred to remediation also would allow tracking 
of students' adherence to placement decisions. Research suggests that many students do not complete developmental 
education seguences because they do not enroll in the course to which they are referred or, upon the completion of the 
first remedial course that they need, do not enroll in the next remedial course in the seguence (Bailey, Jeong, & Cho 2010). 
Improving success in developmental coursework is likely to reguire careful monitoring by systems and institutions of 
students' progress from assessment through referral to, enrollment in, and completion of developmental courses. 

Even if a system collects data on remedial assessments and placement decisions, the institutional and system policies on 
assessment and placement can affect the guality of that data. If colleges do not make assessment mandatory, they may be 
missing data on the level of preparation for many of their students. Additionally, if multiple assessment measures or cut 
scores are used across the colleges, the classification of students who need remedial coursework will be inconsistent across 
colleges, limiting the value of comparisons. To most accurately identify students in need of basic skills, state systems should 
collect data on students' preparation and require institutions to assess all students and encourage greater uniformity in 
assessment tests and cut scores. 

The academic success of financially needy students is of great importance to states, systems, and institutions. 
Unfortunately, determining students' income is difficult and this limits tracking the success of needy students. Whether 
or not a student received a Pell grant is often used as a proxy for income, but the value of this measure depends on the 
number of students applying for the grant. Many low-income students do not apply (Asher 2007). Unless all students who 
qualify for Pell apply and receive the grant, differences between Pell recipients and non-Pell recipients could be due to the 
students' income level or to the effect of financial aid. Using the median income of students' neighborhoods is a possible 
proxy for direct income data but does not fully substitute for accurate student-level income data (Dougherty, Hare, & Natow 
1999). 

Tracking community college students who transfer to four-year private or out-of-state institutions can be a particular 
problem in states that have a large private sector or are located in regions where students frequently cross state 
boundaries. Without good data on student movement into private and out-of-state institutions, transfer rates will be 
understated in accountability reporting and little information will be available about the subsequent progress of transfer 
students. Additionally, colleges and systems will not have the best information to make changes to improve transfer rates if 
students transferring to private or out-of-state institutions are misclassified as failing to transfer. 



5. DATA ARE NOT ENOUGH: 

A CULTURE SUPPORTIVE OF DATA USE MUST BE ESTABLISHED 



Beyond choosing a good set of milestones and success indicators and having the data systems to track them; states, 
systems, and institutions need the capacity to convert data into useful information. They also need the capacity to translate 
this information into policies and practices that benefit students. 7 As states, systems, and institutions begin or expand their 
work with longitudinal data systems, they should also address the human capacity demands of a data-driven decision- 
making enterprise. 

INSTITUTIONAL CAPACITY 

There are several factors related to an institution's capacity to analyze milestones and success indicators, and make 
changes based on this analysis. Institutions need to engage faculty and staff in broad dialogue about the value of data 
and discuss what the data mean. This process should focus on drawing connections between the data and the policies and 
practices that may explain them. To facilitate this process, institutions need the research capacity to produce readable 
reports for broad audiences and to engage in additional, iterative analyses of student-level data to clarify findings 
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CROSS-INSTITUTIONAL 
DISCUSSIONS OF DATA 
CREATE THE OPPORTUNITY 
FOR INSTITUTIONAL AND 
PROFESSIONAL DEVELOPMENT 
THROUGH HIGHLIGHTING 
PERFORMANCE DIFFERENCES 
AND EXPLORING THEIR CAUSES. 



and answer questions. Additional analyses may involve using other student 
enrollment data not included in the core set of milestones and success indicators, 
as well as breaking the data down further (e.g., to the program or classroom 
level). Institutions also need to be committed to making institutional changes 
based on the data. They need to have instilled a culture of evidence-based 
decision making. Institutional leadership is vital for instilling such an institutional 
culture. Effective advocacy for elevating relevant findings to the system and 
state policy levels is another aspect of institutional capacity to use data to 
improve student outcomes. 

POSTSECONDARY SYSTEM CAPACITY 

State systems require analytical capacity of their own to analyze milestones and 
success indicators. Systems can produce sound institutional comparisons by 
providing centralized data storage and ensuring uniform data definitions across 
colleges within the system. System offices can also analyze data at institutional 
and system levels and provide supplementary research support to compensate 
for lower levels of capacity available at some institutions. System offices can 
facilitate conversations across colleges about milestone and success indicator 
data. This can be done through statewide organizations such as presidents' 
councils and academic senates. Cross-institutional discussions of data create the 
opportunity for institutional and professional development through highlighting 
performance differences and exploring their causes. These discussions can 
help colleges identify policies and practices that lead to improved student 
outcomes. Systems can share data with one another, as is done in the Achieving 
the Dream Cross-State Data Work Group and the Access to Success initiative. 
These discussions can help systems identify the likely effects of state and 
system policies on student outcomes. Like institutions, systems need to have the 
commitment to make policy changes and advocate for state policy change. 

STATE-LEVEL CAPACITY 

Capacity at the state level begins, first and foremost, with a framework of state 
goals around which to establish a meaningful state-level reporting system. 
Milestones tied to important state goals are the most useful form of information 
on student outcomes for state policymakers. Thus, it is necessary to have a 
clearly articulated framework of state goals on which to base a set of appropriate 
measures. 

In addition, states need a process for regularly and systematically reviewing 
and interpreting the data. Some states do this effectively by means of a higher 
education accountability system under which a designated coordinating agency 
uses the data to diagnose the state's challenges and make recommendations for 
changes in policy and investment strategies. The role of the coordinating agency 
points to a final state capacity requirement: an effective state coordinating body 
to translate institutional- and system-level findings to the realm of state policy. 
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IN SUMMARY 



The rapidly developing efforts to track milestones and success indicators have much potential for improving student 
outcomes. These approaches can be most effective when they give careful consideration to the appropriate uses for 
the different types of data, how the measures are tracked, and the collection of high-quality data on important student 
outcomes and behaviors, and when processes and structures are developed to utilize the information effectively. 

Through the effective use of these measures; states, systems, and institutions can create a foundation for using student 
data to drive change in policy and practice. Over time, we should see an increase in the attainment of interim milestones 
and, most importantly, an improvement of final outcomes: completion and career advancement across all populations. 
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APPENDIX 1: SUMMARY OF MILESTONES 
AND SUCCESS INDICATORS 



MILESTONES 


SUCCESS INDICATORS 


TYPE OF MEASURE 


WHO IS TRACKING IT 


TYPE OF MEASURE 


WHO IS TRACKING IT 


Retention/persistence 


ATD, CLASS, Complete College 
America, NCCBP, CUNY PMP, 
Tennessee 


Began remediation in a 
specified period of time 


A2S 


Completed or succeeded in 
developmental courses 


NCCBP, Ohio, WA SAI 


Completed remedial courses 
or coursework in a specified 
period of time 


A2S, CLASS, ATD, Gates 


Passed exams on exit from 
remediation 


CUNY PMP 


Completed gateway courses in 
a specified period of time 


A2S, ATD, Complete College 
America 


Remedial student enrollment 
in college-level coursework 


Tennessee 


Credit completion ratios 


A2S, ATD, CLASS, Complete 
College America, Tennessee 


Remedial student completion 
of college-level coursework 


Complete College America, 
NCCBP, A2S, Tennessee 


Basic skills course completion 
ratios 


CLASS 


Completed five credits in 
mathematics 


Ohio, WA SAI 


Completed credits in a 
specified period of time 


A2S, ATD, Complete College 
America, CUNY PMP, Gates 


Completed a certain number 
of credits 


Ohio, WA SAI, Gates 


Completed or enrolled in 
summer courses 


A2S, CUNY PMP 


Completed gateway course 


CLASS, NCCBP, Tennessee, 
CUNY PMP 


Enrolled full time 


A2S, Gates 


Passed proficiency exams 


CUNY PMP 


Enrolled continuously 


A2S 


Still enrolled with a number of 
units completed 


ATD 


Students' grades 


NCCBP 


Completed program objective 


Tennessee 


Declared major at specific 


CUNY PMP 


Completed a degree or a 
number of units 


Ohio 


Passed remedial skills test in a 
specified period of time 


CUNY PMP 


Transferred without an award 


ATD 


High school student 
enrollment in college courses 
or completion of college 
credits 


CUNY PMP, Ohio 


Transferred 


NCCBP, Ohio, Tennessee, 
NCCBP, Complete College 
America 


Student satisfaction 


CUNY PMP, Tennessee 


Completed certificate or 
degree without transfer 


ATD 


Admission or placement test 
scores 


CUNY PMP, Gates 
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MILESTONES 


SUCCESS INDICATORS 


TYPE OF MEASURE 


WHO’S TRACKING IT 


TYPE OF MEASURE 


WHO’S TRACKING IT 


Transferred with a degree, 
certificate, or earned a 
minimum number of units 


ATD, Gates 


Community college student 
enrollment in university 
courses 


Ohio 


Completed certificate or 
degree 


WA SAI, CUNY PMP, 
Tennessee, Complete College 
America, NCCBP, Gates 


Time to degree 


Complete College America, 
Gates 


Completed credential in 
specific fields 


Gates 


Enrolled in remediation 


Complete College America, 
Gates 


Total success rate 


ATD 


Degree credits earned by 
remedial students in a period 
of time 


Gates 


Passed licensure exams 


CUNY PMP 


Remedial students who 
complete college level 
coursework in that subject in a 
period of time 


Gates 


Job placement 


CUNY PMP, Gates 


Enrolling in college soon after 
high school 


Gates 


ATD: Achieving the Dream Cross-State Data Work Group; CLASS: California Leadership Alliance for Student Success; NCCBP: 
National Community College Benchmarking Project; CUNY PMP: City University of New York Performance Management Process; 
Tennessee: Tennessee Performance Funding: Ohio: Ohio Performance Funding: WA SAI: Washington's Student Achievement 
initiative', A2S: Access to Success: Gates: Bill & Melinda Gates Foundation Student Progress and Completion Measures for 
Grantees. 


The 130 colleges funded for Achieving the Dream also collect and report data about their efforts to improve student outcomes on 
a larger set of measures. The Achieving the Dream metrics in this report refer to those developed by the Cross-State Data Work 
Group and include all community colleges in participating states, not just those funded for Achieving the Dream. 



24 TAKING THE NEXTSTEP 




APPENDIX 2: DETAILED DESCRIPTIONS OF 
MILESTONES AND INDICATORS 



MULTI-STATE INITIATIVES 



ACHIEVING THE DREAM CROSS-STATE WORKGROUP 


MILESTONES 


SUCCESS INDICATORS 


Award of less than an Associate's degree w/o transfer 


Earned 24 or more hours in first year 


Award of an Associate's degree or higher w/o transfer 


Completed developmental math by year 2 


Award of less than an Associate's degree and transferred 


Earned 48 or more hours in first two years 


Award of an Associate's degree or higher and transferred 


Passed gatekeeper English or higher by year 3 


Transferred w/o an award 


Passed gatekeeper math or higher by year 3 


Still enrolled with 30 or more college hours 


Passed 80% or more of attempted hours in first year 


Total success rate 




Persisted fall to spring 




Persisted fall to fall 




ACCESS TO SUCCESS 


MILESTONES 


SUCCESS INDICATORS 


Students began college-level work in discipline that 
remediation occurred 


Began remediation in first term 


Student completed college-level course/earned credit in the 
discipline that remediation occurred 


Began remediation after first term 




Completed remediation in first year 




Completed remediation after first year 




Complete gateway courses (English, math) in first two years (2- 
year institution) or first year (4-year institution) 




First-year credit completion ratio 




First-year credit accumulation 




Earned summer credits 




Full-time enrollment-attempted at least 12 credits during the 
first term 




Continuous enrollment 


COMPLETE COLLEGE AMERICA (DRAFT) 


MILESTONES 


SUCCESS INDICATORS 


Percent of freshman who place and enroll in remedial math, 
English, or both, and complete college credit-bearing courses in 
that subject 


Percent of students who complete first college credit-bearing 
math/English in first two consecutive academic years 


Graduation rates for 100%, 150%, and 200% (for Associate's 
degrees) of normal time 


Full-time students who complete 24 credits in first year 
and part-time students who complete 12 credits in the first 
academic year 


Percent of students who transfer from a 2-year to a 4-year 


Number of credit hours completed/credit hours attempted 


Annual number and percentage of certificates of at least one 
year, Associate's degrees, and Bachelor's degrees awarded 


Average length of time in years, and average number of credits 
that graduating students took to earn an Associate's degree, a 
Bachelor's degree, or a certificate 


Fall-to-fall retention 
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SINGLE STATE OR SINGLE SYSTEM INITIATIVES 



MILESTONES 


SUCCESS INDICATORS 


Number of students earning their first 15 semester credit hours 
at that institution in a given year 

Number of students earning their first 30 semester credit 
hours at that institution in a given year 
Number of students who earn 5 college credits in college-level 
computation or quantitative reasoning programs 
Every time a student makes a significant test gain in math, 
listening, or reading on CASAS or earns a GED/HS diploma 
Every time a student completes a level in pre-college English 
and/or math with a qualifying grade 

Earning a certificate backed by at least one year of college, a 
two-year degree, or completion of an apprenticeship 





MILESTONES 


SUCCESS INDICATORS 


Number of students who complete some remedial student 
credit hours at that institution in a given year 


Number of students who enroll in college courses in a given 
year while enrolled in high school 


Number of students who either earn an Associate's degree in a 
given year or at least 45 semester credit hours in a given year 


Number of students who enroll in college courses at a 
university, including branches, in a given year 


One transfer point is awarded for each year of semester credits 
attempted that lead to a transfer to a USO university or branch 
campus in a given year 

Number of students earning their first 15 semester credit hours 
at that institution in a given year 

Number of students earning their first 30 semester credit 
hours at that institution in a given year 
Number of students earning their first 5 semester credit 
hours of General Studies-level mathematics courses at that 
institution in a given year 





MILESTONES 


SUCCESS INDICATORS 


Pass rate in reading on exit from remediation 


Percentage of students passing freshman composition and 
mathematics courses with a grade of C or better 


Pass rate in writing on exit from remediation 


Percentage of freshmen and transfers taking one or more 
courses the summer after entry 


Pass rate in math on exit from remediation 


Percentage of baccalaureate students who have declared a 
major by the 70th credit 


Percentage of required invitees who took the CUNY proficiency 
exam 


Average number of credits earned by full-time, first-time 
freshmen in baccalaureate programs in first 12 months 


Percentage or required test-takers passing the CUNY 
Proficiency exam 


Percentage of non-ESL SEEK students who pass all basic skills 
tests within one year 


Four-year graduation rate for first-time, full-time freshmen in 
baccalaureate programs 


Percentage of ESL students (SEEK and regular) who pass all 
basic skills tests within two years 
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MILESTONES 


SUCCESS INDICATORS 


Six-year graduation rate for first-time, full-time freshmen in 
baccalaureate programs 


Percentage of Associate's degree students not fully skills 
proficient upon initial testing who have met basic skills 
proficiency in reading, writing, and math by the 30th credit. 


Four-year graduation rate for full-time transfers in 
baccalaureate programs 


Total College Now enrollment (high school and college credit) 


Six-year graduation rate for full-time transfers in baccalaureate 
programs 


Percentage of College Now participants who earn an A, B, or C 
in College Now high school and college-credit courses 


Six-year graduation rate for full-time, first-time freshmen in 
Associate's degree programs 


Student satisfaction with academic support services 


Percentage passing the Liberal Arts and Sciences Test (LAST) 
for teacher certification 


Student satisfaction with student services 


Percentage passing the Assessment of Teaching Skills-Written 
(ATS-W) for teacher certification 


Student satisfaction with access to computer technology 


Percentage passing a Content Specialty Test (CST) 


Mean SAT scores of regularly admitted first-time freshmen 
enrolled in baccalaureate programs 


Percentage passing the NCLEX exam 


Student satisfaction with administrative services 


Percentage of test takers with an advanced degree passing at 

least one segment of the Uniform CPA exam 

Six-month job placement rate in career and technical education 

programs 

One-year retention rate for full-time transfers into 
baccalaureate programs 

Two-year retention rate for full-time transfers into 
baccalaureate programs 

One-year retention rate for first-time freshmen in Associate's 
degree programs 

One-year retention rate for first-time, full-time freshmen in 
baccalaureate programs 

Two-year retention rate for first-time, full-time freshmen in 
baccalaureate programs 





MILESTONES 


SUCCESS INDICATORS 


Proportion of students who successfully completed college- 
level courses 


Scores on the Community College Survey of Student 
Engagement 


Proportion of students who successfully completed their 
English Composition 1 course 

Proportion of students who successfully completed their 
English Composition II course 

Proportion of students who successfully completed their 
college algebra course 

Proportion of students who successfully completed their 
developmental course and enrolled in a related college-level 
course 
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TENNESSEE (continued) 



MILESTONES 


SUCCESS INDICATORS 


Proportion of students who successfully completed their 
developmental course and a related college-level course 




Cumulative first-year grade point average at transfer institution 




Proportion of graduates that complete their educational 
objective 




Proportion of leavers and non-completers who achieved their 
educational objective 




Fall-to-fall retention rate 




Degree completion with three or six years 




Average score on a General Education Assessment 




Average score on a major assessment (student learning 
measure) 





MULTI-INSTITUTIONAL INITIATIVES 



NATIONAL COMMUNITY COLLEGE BENCHMARKING PROJECT 


MILESTONES 


SUCCESS INDICATORS 


Success rates in core academic skills areas 


College-level course retention and success rates 


Certificate, degree completion, and transfer rates 


Performance in transfer institutions 


Developmental course retention and success rates 


Institution-wide grade information 


Developmental student success in first college-level course 




Developmental course retention and success rates 




Next-term and fall-to-fall persistence rates 




CALIFORNIA LEADERSHIP ALLIANCE FOR STUDENT SUCCESS 


MILESTONES 


SUCCESS INDICATORS 


Completion of the first collegiate-level courses in English 
composition, U.S. history, and college algebra 


Overall rate of successful basic skills course completion (course 
completion ratio) 


Rate of students who earn degrees or certificates 


Rate of basic skills students completion of any basic skills 
course in first term 


Rate of entering students' participation in basic skills education 


Overall rate of successful course completion (course 
completion ratio) 


Entering students' persistence rate from fall to spring term of 
enrollment 




Entering students' retention from fall to fall term of enrollment 
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